UI evaluators: isolated, model-pinned evaluator runs and start_evaluation - #261

Summary

This is the foundation of UI evaluators, the first of four stacked PRs for the OpenSpec change ui-evaluators, whose approved mockup is in openspec/changes/ui-evaluators/ui/mockups.html. A coordinator can now start a blind evaluator run on one exact model, with only a prompt and a run folder. It is woken with one short line when the run ends. The app driver and the desktop driver come in the stacked PRs that follow.

• Evaluator runs. This adds a new launch kind and an evaluations table. A run belongs to a project, never to a ticket or worktree. Every run starts a fresh session, which is archived when it ends.
• Provider-enforced isolation. A run has no shell, web access, coordination tools, brain or repository. It reads and writes only its run folder, and its first message is exactly prompt.md.
  • Claude runs with -p, --setting-sources "", a strict empty MCP config, dontAsk, Read,Glob,Grep,Write,Edit limited to the folder, and auto memory off.
  • Codex runs exec --ignore-user-config --ignore-rules with a workspace-write sandbox at the folder, no network and no web search. It gets its own CODEX_HOME, which holds only a link to auth.json, and an empty HOME, so neither ~/.codex/AGENTS.md nor the user's skills reach it.
  • The environment is cleared to an allowlist, so no WORKSPACE_* variable passes through.
• Exact model. A run accepts only the models the provider's CLI lists, including full IDs, and never uses Who runs what or a fallback. It records the model the provider reports. A full ID that drifts fails the run. At a usage limit the run waits for the reset in the same session.
• Tools. Coordinators get start_evaluation, list_evaluations and stop_evaluation. Refusals name the reason. The end message is only Run <id> <state> · model <m> · folder <f> · <N min>.
• Authorization. Each project can be set to off, app or desktop, and it is off by default. Only the human sets it, through the Inbox card raised on a project's first run. Closing the project turns it off.
• Queue and screen lock. App runs run side by side. Desktop runs go first in, first out with one screen holder; desktop mode is refused until the desktop-driver PR lands.
• Recovery. A run cut short by a host crash or kill fails at the next start, and its coordinator is woken.

Decisions

• Codex rewrites auth.json in place on a token refresh, so linking it from the evaluator's Codex home is safe. This was checked with a probe using fake tokens and a local refresh endpoint; details are in design.md.
• The end message includes the model the provider reported, so model drift is visible. That model is provider metadata, not report text.

How it was tested

• cargo fmt --check and cargo clippy --workspace --all-targets -D warnings are clean.
• cargo test --workspace: 2557 tests pass.
• New tests cover:
  • the Claude and Codex launch arguments and environment;
  • the start and tool-call checks;
  • the refusals;
  • model pinning and drift;
  • waiting at a limit without a fallback;
  • the end message;
  • the queue and screen lock;
  • stop, close and startup recovery;
  • contract tests for the three tools.
• The ignored live tests in tests/evaluator_isolation.rs ran against the real Claude and Codex CLIs. Neither saw the user's CLAUDE.md, AGENTS.md or skills, and neither could write outside its folder.

Follow-ups

• The evaluator environment allowlist drops the provider's own auth, config and proxy variables (ANTHROPIC_API_KEY, CLAUDE_CONFIG_DIR, HTTPS_PROXY), so a run could fail to start where they are needed (crates/workspace-host/src/provider/evaluator.rs:20).

Reviewed with Wiffletree: 1 full review and 0 re-checks, 2 rounds (Tests, Claude, Codex, Style)

🤖 Generated with Claude Code